feat: third-party eval metrics adapter (DeepEval + Autoevals) with strands-evals mappers - #568

Do we need to accept aws lambda?

Replaced lambda examples with named functions in docstrings

1. We have not updated this type Dict[str, Any]?
2. Regarding naming, "Custom mapper" would be a better choice for describing an arbitrary mapper provided by a customer?

1.The type Dict[str, Any] is correct — Autoevals has no typed
input class like DeepEval's LLMTestCase. Scorers take plain
kwargs (input, output, expected) and the dict gets unpacked
as metric.eval(**kwargs). Different scorers need different
keys, so a generic dict is the right contract.

label: Optional[str] = None label in EvaluatorOutput is optional.
We don't need to have a default value to generate a label when a metric doesn't have a label and the user doesn't provide any threshold. Can we set default as None?

Changed threshold default to None. When no threshold is set,
no Pass/Fail judgment is made. Note: EvaluatorOutput's
validator currently requires a label for success responses —
I'm returning the score as the label string when threshold is
None. Let me know if you'd rather relax the validator to
allow label=None.

this naming is not aligned with DeepEval adaptor. Looks like you haven't updated AutoevalsAdapter.
If you haven't completed end to end tests for AutoevalsAdapter, you should not include it in your PR.

Working on E2E test plan now. Considering how to strcuture parameterized Lambda handlers and how to obtains span from strands, openinference, opentelemetry, and customer mappers.

using lambda? same as above

fixed

naming? same as above

fixed

Attributes:
context: Span context for the entry, e.g. {"spanContext": {"sessionId", "traceId"}}.

Why do we use the first reference_input? If all the reference_inputs have an empty context or an empty spanContext, using the first one as the default makes sense. But what if the sessionId/traceId has been explicitly specified?
reference for SpanContext : https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_SpanContext.html

    "evaluationReferenceInputs": [{
            "context": {
                "spanContext": {
                    "sessionId": "a9f5f0c7-3c94-43cc-933d-7e191caf9d8d"
                }
            },
            "assertions": [{
                    "text": "search_flights tool call args are origin=SEA and destination=NYC"
                },
                {
                    "text": "search_hotels is called with city=NYC"
                }
            ],
            "expectedTrajectory": {
                "toolNames": ["search_flights", "book_flight", "search_hotels", "book_hotel"]
            }
        }, {
            "context": {
                "spanContext": {
                    "traceId": "69b0693f0ed8ff777d364fe4265fe2a4",
                    "sessionId": "a9f5f0c7-3c94-43cc-933d-7e191caf9d8d"
                }
            },
            "expectedResponse": {
                "text": "Booked flight DL420"
            }
        },
        {
            "context": {
                "spanContext": {
                    "traceId": "69b0696718e27bea18ac5a74148a81d8",
                    "sessionId": "a9f5f0c7-3c94-43cc-933d-7e191caf9d8d"
                }
            },
            "expectedResponse": {
                "text": "Booked the plaza hotel for $1350"
            }
        }
    ],

Fixed — now matches reference_input to the target trace by spanContext.traceId. If a reference input has a matching traceId, it's used. If no traceId is specified (empty context), it's treated as the default. Falls back to first entry if nothing matches.

Please regenerate uv.lock after this dependency change. It still resolves strands-agents-evals==0.1.0, which does not provide detect_otel_mapper. Frozen installs will fail at import time.

When you use reference_inputs[0], you are only reading one scoped reference. A TRACE evaluation can contain both a session-level reference and a matching trace-level reference, so this can drop either expectedResponse or the session-level assertions and trajectory. Please inspect and combine the relevant entries instead of selecting one by position.

I would remove this skip. Otherwise, CI can pass without making it clear that the test never ran.

Filtering a TOOL_CALL evaluation to only the target span drops the enclosing AgentInvocationSpan. DeepEval's Tool Correctness documentation lists input, actual_output, tools_called, and expected_tools as required arguments. The agent span provides input and actual_output, while the targeted tool span provides the data for tools_called.

For example, suppose a trace contains an invoke_agent span with the input "What's the weather?" and output "72 F", plus an execute_tool span for get_weather. When target_span_id points to the tool span, this filter removes the agent span. map_spans() then raises No AgentInvocationSpan found in session, and the adapter returns FIELD_EXTRACTION_ERROR instead of running ToolCorrectnessMetric.

Successful AgentCore code-based evaluator responses cannot omit label. This change permits EvaluatorOutput(value=0.75, label=None), which is the default path in AutoEvalsAdapter when no threshold is provided.

I verified this against the live service. The Lambda completed successfully and returned:

{"value": 0.75, "explanation": "test"}

AgentCore rejected it with:

InvalidLambdaResponse: LambdaEvaluationSuccessResponse.label: Field required

The same Lambda succeeded when it included "label": "Pass". Please restore the existing label validation and make AutoEvalsAdapter always produce a label, such as by requiring a threshold or restoring the 0.5 default. The existing test_label_required_without_error_code also currently fails because this invalid response is accepted by the SDK model.

The branch does not pass the repository's required lint and formatting checks. Using the exact Ruff 0.12.0 version pinned by CI, ruff check . reports 10 errors (F401, G201, E501, and F841), and ruff format --check . reports four files that require formatting. The CI workflow runs pre-commit run --all-files, so the lint job will fail as submitted. Please run pre-commit run --all-files, address the remaining errors, and rerun it until the branch is clean.